Skip to content

CLI conformance test environment for the OpenStack API and terraform (SPEC-028) - #107

Open
senolcolak wants to merge 47 commits into
mainfrom
feature/105
Open

senolcolak wants to merge 47 commits into
mainfrom
feature/105

Conversation

@senolcolak

@senolcolak senolcolak commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Adds executable compatibility checks for the OpenStack CLI and Terraform provider, and fixes the API behavior those checks exercise. Closes #105.

Changes

  • Add the CLI conformance harness with per-service manifests, JSON/JUnit reports, cleanup, and coverage gates. PR CI uses PostgreSQL and pinned OpenStack CLI 7.0.0; nightly runs include cross-service workflows and strict checks.
  • Add a Terraform wrapper that applies the compatibility module, checks drift, destroys resources, and verifies empty state and no remaining objects.
  • Fix filters, response fields, error envelopes, tenant ownership, router interfaces, application credential roles, and lifecycle behavior across Glance, Nova, Neutron, Keystone, and Cinder. Migrations 076–082 support the schema changes.
  • Integrate main through d81f081 (fix(security): authorize APIs and harden allocation and uploads #104), retaining serialized subnet IP allocation, gateway exclusion, shared fixed-IP handling, and null-safe Cinder descriptions.
  • Preserve duplicate server-group names, matching Flamingo Nova: migration 078 no longer merges/deletes distinct groups. Migration 081 removes the pre-release name-uniqueness rule; 082 persists validated server-group rules. Previously deleted pre-release groups cannot be reconstructed by this migration.
  • Repair CI setup and use a single setup-go cache keyed by both module dependency files. Preserve main's stable Go toolchain selection. Require SAP sign-off only on pull-request events, fixing main's aggregate CI failure when sign-off is intentionally skipped on push.
  • Make evidence gates reject unknown statuses, duplicate case IDs, failed catalog/resource listings, and unreadable Terraform state. Include CLI harness regression tests in PR CI.

Review fixes

  • Commit router port claims and interface records atomically; use existing ports' actual IPs and serialize automatic allocation and subnet deletion with other allocators.
  • Persist application credentials and roles atomically; reject malformed roles and deduplicate aliases.
  • Retain shelve snapshot records until restart and the ACTIVE state update succeed.
  • Preserve snapshot metadata in update responses and count Unicode characters for image/tag/backup name limits.
  • Reject malformed CLI evidence, preserve assertion failures under nounset, and fail contract runs on package errors or zero executed tests.
  • Preserve Terraform recovery state and prerequisite images after failed cleanup, reject missing/invalid post-destroy plan summaries, and avoid inherited OS_CLOUD overriding the selected target.
  • Remove random floating-IP collisions from a test fixture. Add regression coverage for rollback, exhausted pools, duplicate-name preservation, rule persistence, and evidence gates.

Validation

Review/fix passes completed through 54fd0b1; the final pass found no further actionable issues.

  • Full internal/pkg Go suite with SQLite passed after the fixes.
  • make lint: zero issues. make build passed.
  • 15 CLI/contract-runner tests and 7 Terraform-wrapper tests passed.
  • ShellCheck (error severity) and git diff --check passed.
  • Current CI and security are validating 54fd0b1; results pending. PostgreSQL schema parity and live CLI validation run there because no local PostgreSQL server or Docker daemon was available.

Known limitations

  • The known-failure baseline covers two documented cases: console URLs in stub mode, and backup lookup by name in pinned OSC 7.0.0. The gates retain them as visible known failures and reject new unbaselined failures.
  • Token revocation retains the existing JWT behavior: a valid token receives 204 and expires naturally; there is no denylist.
  • Full nightly CLI and live Terraform runs were not rerun. Terraform evidence gates are covered by regression tests with fake external clients.

Uwe Schwaeke added 30 commits October 15, 2026 23:03
…onment (issue #105)

Analysis and implementation plan for the test-environment enhancement:
verify the built binary against the python OpenStack CLI in all cases,
with a machine-auditable coverage matrix, a hard CI gate, and Terraform
conformance in CI. Also fixes the broken contract-test runner (portable
paths, nova added to the service loop, strict mode) and marks the three
superseded test-status snapshots as historical.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…hase 1)

New test/cli/: osc() wrapper (always -f json, auto-retry for formatter-less
mutation commands, timeout, rc/stdout/stderr capture), jq assertions,
t_begin/t_end case lifecycle with known-failure and skip states, JSON+JUnit
reporting, idempotent fixtures with a crash-safe cli-conf-* prefix sweep,
and bin/run.sh (external server, or binary-test compose stack that runs the
built bin/lightstack artifact). bin/coverage.sh cross-references the
per-service manifests with recorded results — the machine-checkable
definition of 'all cases'; it fails on untested/failed/skipped tier A+B
commands, allows but counts known-failures. Smoke suite verified live:
11/11 pass, rerun-safe, warnings-clean.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…s on create

ListImages silently ignored name, disk_format, container_format, size_min,
size_max, owner and tag, returning unfiltered results. The exact-match
name filter is what terraform-provider-openstack's image data source
relies on server-side (no client-side matching), so any server holding
more than one image broke data lookups with 'Your query returned more
than one result' — found by the SPEC-028 terraform conformance run. The
same gap made 'openstack image list --tag/--disk-format' vacuous.

CreateImage echoed the requested tags in its 201 response but never wrote
the image_tags rows, so a follow-up show returned tags: [] —
'openstack image create --tag' silently dropped them.

Found by the SPEC-028 CLI conformance suites (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
- os-server-groups: accept the microversion 2.64+ 'policy' string (and
  the legacy 'policies' array) and answer with both fields. The server
  advertises compute 2.90, so every 2.64+ client — OSC 7 included — was
  rejected with 400 on server group create.
- os-aggregates: a non-UUID path reference (openstacksdk's find_aggregate
  probes by name first) hit the uuid cast error and answered 500 instead
  of the 404 the client needs to continue its lookup.
- createBackup: name the backup image exactly the requested name
  (upstream passes it through verbatim); the -<type>-<timestamp> suffix
  broke every client that looks the image up afterwards. Rotation still
  finds candidates via image_tags.
- shelve/unshelve: unshelve now consumes the instance_snapshots row, and
  shelve reclaims any stale one, fixing the shelve -> unshelve -> shelve
  500 on UNIQUE(instance_id).
- server show/list detail now carry 'locked' (exposed since
  microversion 2.9 upstream); the actions maintained the column but the
  responses never surfaced it, so 'openstack server show' printed null.

Found by the SPEC-028 CLI conformance suites (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…standard floating-ip list filters

GET /v2.0/networks?tenant_id=<project> ANDed the caller's read scope
into the tenant filter, so an administrator could never see another
project's private networks — 'network list --project <id>' answered []
even though the project owned networks. Real Neutron honours the
filter for admin tokens: when the caller is an admin, the tenant
filter is now the authoritative owner scope. Non-admin callers keep
the scope intersection (their own, plus shared/external).

GET /v2.0/floatingips ignored every list filter — floating_network_id,
port_id, router_id, status, fixed_ip_address, floating_ip_address,
description — so 'floating ip list --network <net>' returned addresses
from every network. All standard exact-match filters are now applied
after the project scope; a non-UUID value on a UUID column answers the
same empty list an unknown id does, not a 500.

Found by the SPEC-028 CLI conformance suites (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
- user show/list now select email and description; both were accepted on
  create/set but silently dropped on read, so 'user show' printed
  email: null for a user created with one.
- role create maps the UNIQUE(name) violation to 409 instead of 500.
- role assignment list honours include_names=true, embedding user.name,
  user/project domain names and project names — openstacksdk's consumer
  dies with KeyError 'name' on the bare-id shape, so 'role assignment
  list --names' exited 1.
- domain create/show responses carry links.self; the pinned OSC 7
  formatter aborts with KeyError 'links' without it.
- application credentials: a role given by name only panicked on a nil
  type assertion (gin recovered it as 500 after the row was already
  committed); roles are now resolved by id or name before anything is
  written, an unresolvable role is a 400, the duplicate (user_id, name)
  answers 409 instead of 500, and list/show always carry the roles key.
- migration 074 finally creates application_credential_roles, which the
  handlers have referenced since 029 introduced the feature: every roles
  lookup failed, the error was swallowed, and the roles key vanished
  from the responses.
- token revoke validates the subject token and answers 404 when it
  names nothing revocable (204 for every request hid typos and expired
  ids); a valid token is still acknowledged with 204 — JWTs expire
  naturally, there is no session to destroy.

Found by the SPEC-028 CLI conformance suites (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
- volume set accepts the legacy v2 field names (display_name /
  display_description) that python-cinderclient still PUTs for
  --name/--description; the v3 names keep precedence when both are sent.
  Only the v3 names were bound, so the update succeeded but persisted
  nothing.
- volume show and list-detail serialize the volume's user metadata; the
  object answered {} however much was set, so 'volume show' always
  reported empty properties.
- os-set_bootable implements 'volume set --bootable/--non-bootable'
  (was: 400 unknown action); bootable is attach metadata, so it is
  accepted in any status, and malformed payloads 400.
- snapshot create/show/list carry the metadata key — OSC 7's formatter
  crashes with KeyError 'metadata' on a snapshot without one — and show
  and list-detail serialize the snapshot's actual user metadata.
- volume type objects carry their extra_specs, so 'volume type show'
  reports the properties that were set instead of {}.
- a volume type created without a description (NULL) made
  GET /v3/types fail with 500 'failed to read results' for every caller
  until the type was deleted; description is now scanned nullably.
- snapshot, volume-type and backup lookups answer 404 for a non-UUID
  reference instead of surfacing the uuid cast error as 500 — backups
  matter most: cinderclient resolves show/delete by name with no list
  fallback, so the name itself reached the lookup.

Found by the SPEC-028 CLI conformance suites (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
os-force-delete is not implemented server-side, so 'volume delete
--force' answers 400 for every volume — including the in-use leftovers
the sweep exists to reclaim. _delete_all now takes a list of delete
templates and tries them in order, and the volume sweep passes plain
delete first with --force as the fallback.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…orm runs

- cli-conformance job: runs the build artifact's binary natively against
  a PostgreSQL service (no docker), pins the CLI with
  test/cli/constraints.txt, runs the fast subset with the hard coverage
  gate, and uploads the JSONL/JUnit results on any outcome. Added to the
  status-check needs so a regression fails the PR.
- nightly workflow: the full suite with workflows (tier C) under
  --strict, and an apply+destroy run of the unmodified
  terraform-provider-openstack compat module. Both jobs guard on the
  upstream repository so forks can still workflow_dispatch.
- dropped the dead pip install in integration-tests and replaced the
  stale e2e TODO with a pointer to the suites that implement it.

SPEC-028 phase 2 (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
… live server

test/terraform/run.sh wraps the terraform-provider-openstack compatibility
module with a self-contained preflight and a deterministic lifecycle:
per-run tf-cirros-* image, mktemp workspace, TF_PLUGIN_CACHE_DIR,
init/validate/apply, a post-apply no-drift check, destroy, and a final
assertion that the state is empty and no tf-compat-* resources remain
(an adds-only plan proves the destroy reclaimed everything).

The compat module itself stays byte-identical — it is the fixture under
test, not the test.

SPEC-028 phase 3 (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…nd cross-service workflows

Five suites plus their per-service manifests, on the harness core from
the previous commit. Tier A asserts the happy path and response shape of
every manifest command, tier B the negative paths (unknown ids, invalid
payloads, conflicts), tier C the workflows that cross services
(boot-from-volume, snapshot→image→server, attach/detach, security-group
lifecycle, floating-IP association, project isolation, quota
enforcement).

Every case is rerun-safe and self-cleaning: resources are named
cli-conf-<svc>-* and reclaimed by the case that created them; floating
IPs are released explicitly because they carry no name. Known failures
probe for their divergence and record it with evidence — they flip to
hard assertions automatically once the product closes the gap (the
compute suite's server-group, aggregate and backup cases now assert the
fixed behaviour outright).

SPEC-028 phase 1 (issue #105).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
The job drives 'make test-terraform', whose preflight needs the
openstack CLI on PATH; the job built the server but never installed the
CLI, so every scheduled run would have died in preflight.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
'bin/' in .gitignore matched at any depth, so test/cli/bin/run.sh and
coverage.sh were never committed — the ci.yml cli-conformance steps and
make test-cli/cli-coverage/test-all all fail on a fresh checkout. The
pattern now re-includes test/cli/bin/ (scripts, not binaries).

Found by the SPEC-028 review pass.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…28 pass

- floating-ip list filters compare uuid-typed instead of ::text, so an
  upper-case filter value matches like any client expects.
- role-assignment include_names rows without a user or project (not
  creatable through the API today) no longer render a links object with
  empty path segments.
- volume-type extra_specs scans as any-valued jsonb: nothing writes a
  non-string value today, but a future writer would have failed every
  scan of that row.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
- The EXIT trap flattened early failures to exit 0: _cli_cleanup ran rm
  before _standalone_exit captured $?, so an unknown option, a missing
  dependency (preflight exits 2), or any failure before run.sh installs
  its own trap exited successfully — CI would have passed. The exit
  status is now captured first and threaded through explicitly.
- _xml_escape used ${var//pat/repl} with '&' replacements: bash >= 5.2
  enables patsub_replacement by default, so '&' expanded to the matched
  text and corrupted the JUnit entities. Rewritten as a per-character
  case, portable across every bash that can run the harness.
- The keypair sweep deleted nothing: OSC's keypair list has no ID
  column, so the id extraction emitted null for every row. The sweep
  now falls back to Name when a kind has no id — which is also the
  handle 'keypair delete' takes.

Also: --strict (known-failing and skipped cases fail the run — the
nightly's stance), a usage-hint instead of bash's unbound-variable
error for value-less options, and OSC_OUT/OSC_ERR's cross-file contract
is annotated for shellcheck.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
ubuntu-latest is PEP 668 externally-managed, so a bare pip install of
python-openstackclient fails; both conformance jobs now build a venv
and put its bin dir on GITHUB_PATH (the harness resolves the CLI from
PATH). The cli-conformance timeout rises to 40 minutes to match the
suites' measured runtime, and the nightly's full run passes --strict:
known-failing and skipped cases fail the nightly so the debt cannot
silently grow back.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
The wrapper worked in a mktemp workspace that its EXIT trap removed,
so a failed run took its apply/destroy logs with it and the CI
artifact step had nothing to upload. Failure logs (and crash.log) are
now copied to test/terraform/results/ before cleanup; nothing is
written on success.

The residue check gains subnets and routers, and the nameless
floating IPs are verified precisely: their ids are captured from the
state before destroy and each is asserted gone afterwards, instead of
counting every FIP on the server (which other suites may legitimately
hold mid-run).

Also updates the apply-failure hint: the Glance ?name= filter
divergence it described is fixed by this branch.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
- server lock/unlock: assert the locked flag in server show (surfaced
  by the SPEC-028 nova fix) instead of documenting its absence.
- server-group fixtures create with the current 2.64 client shape —
  the 2.15 microversion pin was a workaround for the fixed policies
  binding.
- aggregate purge comment no longer cites a fixed known failure.
- volume snapshot create asserts status 'creating' — upstream Cinder
  also answers 202 'creating'; the transition to 'available' is
  asserted via the list, as before.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…uite

Every case the new network suite recorded as known-failing, fixed:

- GET/PUT /v2.0/ports/{name} answered 500 (uuid cast) instead of 404,
  which breaks openstacksdk's name fallback for 'port show/set/delete'.
- Same 500-not-404 on GET/PUT /v2.0/subnetpools/{name}.
- add_router_interface ignored port_id entirely ('router add port' read
  the empty subnet_id and answered 500); the port path now attaches a
  caller-owned, unclaimed port and derives the subnet from its fixed IP.
- A duplicate interface was a 500 on the (router_id, subnet_id) unique
  constraint that LEAKED the port it had already inserted — which then
  blocked network deletion with a 409 only 'port delete' could recover
  from. The pre-check refuses with 409 before anything is written, and
  the record insert takes its port back if it fails.
- DELETE /v2.0/subnets/{id} succeeded while a router interface held the
  subnet, orphaning the interface port; now upstream's 409 SubnetInUse,
  via jsonb containment on the holding ports' fixed_ips.
- networks/subnet_pools had no description column (migration 075), so
  'network/subnet pool set --description' was a 400 'no updates
  provided'; the column is bound on create and update and emitted on
  every read path.
- 'subnet set --no-dhcp' PUT enable_dhcp, which the update handler had
  no binding for: a 200 that persisted nothing.
- POST /v2.0/security-groups omitted security_group_rules, so OSC
  rendered null and its table formatter crashed ('NoneType' object is
  not iterable); upstream reports [].
- POST /v2.0/subnetpools stored a malformed prefix as-is ('not-a-cidr'
  -> 201); now the caller's 400, as subnets already were.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
30 manifest commands across networks, subnets, ports, routers,
security groups (and rules), subnet pools, RBAC policies and floating
IPs; 41 cases (29 tier A happy-path/shape, 12 tier B negative .neg).
All resources are cli-conf-nt-* and released on every path, including
the nameless floating IPs.

The nine cases that recorded product divergences as known-failing
probes now assert the fixed behaviour outright; two consecutive
standalone runs are 41/41 pass with zero cli-conf-nt-* residue.

One probe fix along the way: jq's alternative operator treats false
as falsy, so '.enable_dhcp // empty' printed nothing exactly when the
flag was off — the one answer the probe existed to detect. tostring,
not // empty.

Commands the CLI itself cannot drive are excluded with reasons in the
manifest description (address group / network flavor: no routes;
'floating ip show': OSC 7.0.0 formatter crash; 'network rbac set':
openstacksdk client-side crash; 'network agent show': no agents in
the deployment).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Distinct from coverage.sh --strict (which additionally requires tier C):
run.sh --strict makes known-failing and skipped cases fail the run —
the nightly's stance, so the debt cannot silently grow back.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
- GET /v3/domains items lacked links.self. The direct by-id GET had it
  (added earlier this branch), but the CLI resolves a domain BY NAME
  through the list (show/delete fall back to ?name= after the by-id GET
  404s), so 'domain show <name>' and 'domain delete <name>' aborted
  with KeyError 'links' anyway. Every list item carries it now.
- Keystone's error envelope was the compute shape
  ({"forbidden": {"message": ...}}). openstacksdk tolerates that, but
  the identity client is keystoneauth1, whose from_response parses only
  {"error": {code, message, title}} (or the API-SIG {"errors": [...]}):
  every other body degrades to 'Unrecognized schema in response body.',
  discarding the reason — 'domain delete <enabled>' told the operator
  nothing about disabling it first. The identity engine now marks its
  requests (the same pattern as Neutron's envelope, SPEC-024) and
  renders upstream's shape; every keystone error reason now reaches
  OSC, keystoneauth and openstacksdk clients alike.

Also re-attributes the one remaining volume known-failure to its real
owner: 'volume backup show <name>' is a pinned-OSC-7.0.0 client bug
(raw sdk get_backup, no name fallback — delete uses find_backup), fixed
client-side in OSC 7.3.0; the server's 404-for-a-name matches real
Cinder. The probe self-heals when the pin is raised.

Full-suite effect: keystone 76/76 (domain.show healed), volume 37/38
with the one documented client-side divergence.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…e docs

The final pre-PR review's findings, all fixed:

- nightly.yml terraform-conformance installed the CLI into the system
  python — PEP 668 kills that on ubuntu-latest, every night. It now uses
  the same venv pattern as the other two install sites.
- test/terraform run.sh's floating-IP residue check probed with
  'floating ip show', which the pinned OSC 7.0.0 fails client-side on
  EVERY invocation ('Invalid formatter provided.') — the check could
  never fail, so a leaked FIP read as a clean destroy. Resolved through
  'floating ip list' + grep-by-id now.
- AddRouterInterface left the caller's port claimed (device_id set, no
  interface row) when the interface-record insert failed; the rollback
  now un-claims it, restoring the pre-call state.
- Glance answered 500 (VARCHAR(255) overflow) for an over-long image
  tag where upstream's schema answers 400; validated on both the create
  and tag-PUT paths, before any write, with a contract test.
- Harness robustness: coverage.sh value-taking options now guard
  against a missing argument (set -u made it an unbound-variable error
  naming $2, not the option) and the dead STRICT variable is gone;
  report.sh pins LC_NUMERIC=C so printf '%.3f' cannot emit a decimal
  comma into the JUnit XML; suites can register an EXIT-trap cleanup
  hook (suite_cleanup), and compute.sh's temp public-key file uses it
  instead of leaking on a mid-flight kill.
- Stale docs and dead code this branch itself made stale: the network
  suite's inverted 'port survives' comment (router remove port deletes
  it — upstream too), the by-name 500 references in network.sh
  (now 404s), the manifest description still listing the nine fixed
  divergences as known-failing, keystone.sh's 'domain create aborts
  client-side' comment, the terraform README's fixed Glance ?name=
  divergence, the no-op probe block in the keystone envelope test, and
  a leftover stacked doc comment in neutron's network list filter.

Re-verified after the fixes: full run 249 pass / 0 fail / 2
known-failing (console stub-mode, pinned-OSC backup show-by-name),
coverage gate PASSED in both plain and --strict modes, glance/keystone/
neutron/common packages green.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
The delta review of 9592cfe confirmed every functional fix correct and
found one leftover plus three nits, all doc-level:

- network.sh's subnetpool-create negative case still said the by-name
  GET 'answers 500 instead of 404' — fixed since ccc4574; now says 404.
  (The four t_known_failure reason strings that still describe the old
  500s are the self-healing probes' diagnostic payloads: they only fire
  again — accurately — if the fix regresses. Left in place, and the
  manifest now says exactly that instead of 'plain assertions'.)
- the terraform wrapper's apply-failure hint pointed at README 'Known
  divergences', which now reads 'None currently' — a circular pointer.
  The hint is self-contained; the pointer is gone, and the README now
  says the hint was reworded rather than removed.
- TestImageTagLengthValidated claimed 'a maximal tag stays legal on
  both paths' but only asserted the PUT; the create path asserts 201
  at 255 chars now too (with cleanup).

Terraform conformance re-verified against the rebuilt binary after
round 2: 9 added / 9 destroyed, zero tf-compat-* residue, PASS —
exercising the reworked FIP residue check live.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…ness polish

All thirteen findings from the fresh full-PR review, fixed:

- nightly cli-conformance-full ran on a 30-minute budget while the
  SMALLER per-PR --fast job had 40 — a strict superset with less room.
  60 minutes now (terraform-conformance: 45).
- the terraform wrapper overwrote a caller's OS_PROJECT_DOMAIN_NAME
  with the user-domain fallback (copy-paste of the line above); it now
  falls back user-domain -> Default only when unset.
- GetNetwork kept the caller's read scope unconditionally, so an admin
  could list another project's private network (--project, fixed
  earlier this branch) and then get a 404 showing the id it just
  listed. Upstream's get_network policy is admin_or_owner or shared or
  external; the admin carve-out matches the list filter, with a
  contract test (TestGetNetworkAdminCarveOut_DB).
- AddRouterInterface's subnet lookup was unscoped: a router owner
  could attach an interface to another project's private subnet by id,
  while the port path was already project-scoped. The subnet row must
  now be the caller's or sit on a shared/external network — 404
  otherwise, keeping its existence private
  (TestAddRouterInterfaceSubnetScope_DB).
- UpdateSubnet applied name/gateway and the DHCP flag in two separate
  statements; a failure between them left a half-applied update. All
  supplied columns land in one UPDATE now (SQLUpdate), preserving the
  documented empty-update 200 semantics.
- NULL email/description serialized as "" where every OpenStack API
  emits null (keystone users/projects across list/show/update paths,
  cinder volume types); new common.NullString renders sql.NullString
  the upstream way.
- run.sh trapped INT/TERM directly, so a killed run could exit 0 — the
  signal traps now re-exit with 130/143 so the EXIT trap propagates
  the right status.
- image.sh's temp dir leaked on early exit; it registers the
  suite_cleanup hook (like compute.sh's key file) instead of an
  end-of-file rm.
- the postgres pin in the CI workflows drifted from the repo's
  deployments (18.3 vs 18.4); all four CI references and
  docker-compose.binary-test.yml now say 18.4-alpine, matching what is
  deployed.
- stale annotations this branch itself outgrew: workflows manifest
  still advertising the fixed admin tenant filter as known-failing;
  keystone.sh's user-set case not asserting the --email round-trip it
  can now perform (and dropping the comment that said it couldn't);
  volume.sh's snapshot/type helper comments and the type-create nodesc
  case header still describing fixed 500s.
- gofmt on the three files the reviewer's -l flagged.

Verified: go vet + all internal packages green (-p 1), gofmt clean,
bash -n/shellcheck/YAML clean; full CLI run + coverage gate re-run on
the rebuilt binary (in flight at commit time, zero failures through
the volume suite).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Round-5 delta review of a1e0ea6: all thirteen fixes verified correct,
but the GetNetwork admin carve-out introduced a new hole of its own.

The network cache key names only (project, network), and the lookup
precedes any visibility check — safe while every cached entry had
passed the project's read scope, which is purely project-based. The
carve-out made visibility role-dependent and left the unconditional
TrySet in place: an admin fetching another project's PRIVATE network
(the exact flow the carve-out enables) cached it under the project's
key for the TTL, and a plain member of that project then read the full
body — owner, subnets and all — as a 200 where the uncached answer is
404. Any cache-enabled deployment (the default compose sets
LIGHTSTACK_CACHE_ENABLED) was exposed.

GetNetwork now caches only rows the project is entitled to by the read
scope (owner OR shared OR external); an admin's read of a foreign
private network is served but never cached, so the key's documented
invariant holds again. The guard is written test-first:
TestGetNetworkCacheDoesNotCrossRoles_DB runs against a real local
Redis (skipped when absent), reproduces the poisoning as a failing
member GET before the fix, and passes after — while proving the
entitlement-positive path still caches (two member reads of the
project's own network).

Also from the same review: AddRouterInterface's new subnet predicate
omitted the network-owner disjunct that subnetReadScope and the
subnet update/delete entitlements grant — the owner of a network could
not attach a subnet an admin created on it, inconsistent with every
other subnet surface. 'n.project_id = $2' joins the predicate, with
the case pinned in TestAddRouterInterfaceSubnetScope_DB.

Verified: go vet + all internal packages green (-p 1), gofmt clean;
live server rebuilt from HEAD and smoke-checked (auth, network and
router list).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Round-5 follow-ups on f4b9ebb, both non-blocking:

- TestGetNetworkCacheDoesNotCrossRoles_DB skipped everywhere but a
  developer machine with local Redis: the unit-tests job had no redis
  service. One redis:7-alpine service (127.0.0.1:6379, redis-cli ping
  health check) makes the poisoning regression actually run in CI.
- The test's 'still caches' half was weak — two 200s would pass with
  the TrySet deleted outright. It now also reads the entry back
  through the cache and checks it holds the right network id, so the
  guard is pinned from both sides: foreign-private rows never cached,
  entitled rows always cached.

Verified: go vet, gofmt clean, full neutron package green against the
local DB and Redis (test runs, not skips).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Final PR-state review: keystone.token.revoke.neg still carried a
conditional t_known_failure claiming 'DELETE /v3/auth/tokens always
returns 204, so the CLI exits 0' — but RevokeToken has answered 404
for an unknown token since cfacd1b (the 204 no-op remains only for
valid tokens, the documented divergence). The branch could never fire;
the case now asserts the non-zero exit the way every other neg case
does (_ke_assert_failed).

Verified by the full run at this HEAD: 249 pass / 0 fail /
2 known-failing / 0 skipped; coverage gate PASSED (233 rows,
0 untested).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…nable

Round-7 harness review found the coverage gate and the failure reporter
could pass vacuously: an empty or truncated results.json (a partial write
from a killed runner) or an unparseable manifest made every jq iterate
over nothing and print 'Coverage gate PASSED'. Every jq feeding the gate
is now exit-checked, a zero-row gate is an error, results.json is written
atomically (tmp+mv on success), and report_failed counts a missing or
unparseable results file as a failing run. Fixing the last exposed a
long-standing ordering bug: run.sh's EXIT trap deleted the results file
BEFORE consulting it, so exit-time failure detection was dead code —
report_failed now runs before the cleanup rm and genuinely gates the exit.

Standalone suite runs after a teardown sweep re-ensure the shared
fixtures via fixtures_require (three cheap show calls; the full
sweep-then-create ensure only when something is missing), so
volume/compute/image/workflows no longer fail their fixture-dependent
cases with confusing missing-project errors — verified by a standalone
volume run right after a sweep: 38 cases, 37 pass, 1 known-failing,
exit 0.

Also: --fast --suite combinations that select nothing now die instead of
exiting 0 green (the vacuous-green class this harness exists to prevent);
the workflows FIP-bound assertion requires the row to exist instead of
passing on absence; the nightly's job guard actually allows the manual
fork dispatches its comment promises; the terraform wrapper's residue
comments no longer claim router/FIP coverage the compat module doesn't
have (the checks stay as forward-looking defenses) and its README lists
subnets; assorted doc-accuracy fixes (fixtures_sweep scope, md5sum
requirement, --suite/--no-cleanup wording). The usage-list case documents
why it deliberately has no length guard: every case before it deletes its
server, so an empty listing is the legitimate steady state.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…hes, races

Fresh-eyes review of the product diff; every finding was reproduced
against the live server before the fix.

Router interfaces (neutron):
- add_router_interface 500'd on a subnet with no gateway: the lookup
  scanned gateway_ip into a plain string and NULL cannot land in one.
  It now scans *string and falls back to the same allocator CreatePort
  uses (upstream attaches with an allocated address, never 500s).
- The port claim is now conditional (WHERE device_id IS NULL): two
  concurrent attaches of one port to different routers both passed the
  read-check and both claims won. A unique-violation on the interface
  insert answers the same 409 as the pre-check instead of a 500, and
  the rollback now also serves that race (it had already claimed the
  port when it loses).

Admin scoping symmetry (neutron): the GetNetwork admin carve-out
stopped at networks — an admin could show a foreign private network
and then 404 on the subnet ids its own body lists. GetSubnet/GetPort
carry the same carve-out; the subnet/port/network listings drop the
read scope for an admin filtering by tenant (and by network/device,
both explicit cross-project pointers), their markers resolve under the
listing's own scope (an admin's tenant-filtered walk used to 404
mid-page on the rows it just returned); the floating-ip and router
listings honor the admin tenant filter (previously silently ignored —
the admin's rows were presented as the target project's).

Cinder: volume list --all-projects/--project implement cinder's
cross-project semantics (admin-only; ignored for non-admins, as nova's
all_tenants is). A NULL volume description renders as JSON null, the
value upstream emits, instead of "".

Glance: image add/remove tag now evict the image and list caches —
GetImage caches the object including its tags, so a tag added with the
dedicated PUT stayed invisible to image show for the cache's whole
TTL. CreateImage wraps image+tags in one transaction (a tag failure
used to 500 with the image committed; the client's retry duplicated
it) and validates name length (VARCHAR(255) overflow was a 500).

Nova: a duplicate server-group name answered 200 with a second row
(upstream 409, and show-by-name became ambiguous). The handler
pre-checks and maps unique violations to 409; migration 076 folds any
existing duplicates (merging members into the earliest survivor) and
adds the constraint migration 031 always intended but never applied
(its CREATE TABLE IF NOT EXISTS has been a no-op since 017 created
the table). createBackup validates the backup name for the same
VARCHAR(255) reason.

Keystone: the identity error-envelope marker is applied at ENGINE
level too — NoRoute/NoMethod under :35357 rendered the compute
envelope and keystoneauth1 degraded it to 'Unrecognized schema'.

Input hardening: the port listing's uuid-typed filters and the
networks tenant filter compare as ::text, so a non-UUID query value
narrows to nothing instead of failing the cast with a 500 (the guard
the floating-ip loop already carried).

Deliberately unchanged: migration 075's TEXT description columns (a
lenient superset of upstream's VARCHAR(255), with no failure mode) and
snapshot descriptions (every writer binds a string, NULL is
unreachable and "" echoes storage).

Verified at this HEAD: full run 249 pass / 0 fail / 2 known-failing /
0 skipped; coverage gate PASSED; full internal Go suite green; four
new DB-backed contract tests pin the router-interface NULL gateway,
the subnet/port carve-out, the admin tenant filter (neutron router +
cinder volume), and the server-group 409.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Uwe Schwaeke and others added 11 commits October 16, 2026 19:57
…caller checks

Delta review of the round-7 harness fixes found four minors, all fixed:

- The zero-suite exit 2 fired AFTER the compose stack could have been
  started — the selection (pure flag logic) now runs BEFORE server
  selection, so an empty selection dies without leaking a stack.
- fixtures_require now probes the subnet too: the sweep deletes subnets
  before networks, so an interrupted sweep leaves exactly 'network
  present, subnet missing' — a state the three probes called healthy
  while compute's server cases failed later. FIX_SUBNET_ID is populated
  on the fast path.
- Both report_write callers check its return (a failed aggregate no
  longer relies on report_failed's fail-closed backstop alone), and the
  JUnit XML is written atomically like results.json. A failed
  results.json write already skips the JUnit block (return 1 first).
- Doc accuracy: the teardown docstring's covered list matches the sweep
  (FIPs are nameless and NOT swept; snapshots, backups, server groups,
  flavors, users, groups are); each suite's fixture comment names
  exactly what that suite references; the zero-suite error prints the
  actual selection () instead of an unset variable.

The delta review's verdict was SHIP with these as fast-follows; its
verification of all eight round-7 fix areas is recorded in the PR.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…guard

The round-8 delta review verified every round-7 product fix correct at
runtime (placeholder counts, scope decisions, migration 076 on a scratch
DB, tests proven to bite on revert) and found four more, all fixed:

- cinder's detail listing stamped every row with the CALLER's project —
  correct by coincidence before the cross-project listing existed, but
  the admin's --project/--all-projects walk now labeled project X's
  volumes as the admin's (os-vol-tenant-attr:tenant_id is consumed by
  `volume list --long --all-projects`). The listing and GetVolume now
  select and render the row's own project, and GetVolume carries the
  admin show carve-out so the walk's rows don't dead-end their
  follow-up show with 404.
- neutron ports had the same caller-as-tenant_id labeling in list and
  show; both render the row's own project now.
- GetRouter and GetFloatingIP carry the admin show carve-out too (the
  same class: their listings cross projects for admins since round 7).
- allocateIP PANICS on a non-positive rand bound: IPv4 /29-/30 and
  every IPv6 subnet with hostBits >= 64 (where the int64 shift
  overflows to 0) turned a NULL-gateway router attach — and any
  CreatePort on such a subnet — into a 500 via the recovery
  middleware. Tenable pools are refused with the empty string the
  callers already treat as 'no usable address'.

Also removed the dead `images:*` eviction patterns (only
image:<project>:<scope>:<id> keys are ever written — imageCacheKey;
four no-op TryDeletePattern calls, two of them added in round 7).

New pins: Redis-backed add/delete-tag cache-eviction tests, the
glance name-length 400, the allocateIP guard (pure), tenant_id
assertions on the cross-project cinder and port responses, and the
router/volume show carve-outs (admin 200 / member 404).

Verified at this HEAD: full run 249 pass / 0 fail / 2 known-failing /
0 skipped, exit 0; coverage gate PASSED; full internal Go suite green.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…rts its guest

Round 9 reviewed the round-8 fixes themselves and found two real defects
in them, a product gap the round-8 guard newly exposed, and one product
bug a full run caught by losing a race — all fixed:

- run.sh's report_write failure check was dead code: '|| failed=1' wrote
  a GLOBAL (the 'local failed=0' came later and re-initialized it), so
  an unwritable results dir with a healthy JSONL still exited 0 green.
  The declaration is hoisted above the call and verified with an
  isolated replica of the exact structure.
- TestAddImageTagEvictsCache was vacuous: the tag was a substring of
  the image name, so the assertion matched every response, tagged or
  not — proven by deleting the eviction and watching the test pass.
  The tag is now distinct and the assertion is the exact tags array;
  both eviction pins now fail with the eviction removed.
- CreatePort persisted fixed_ips with a BLANK ip_address for pools the
  round-8 allocateIP guard refuses (/29-/30, the IPv6 overflow) — the
  500 had become silent bad data. It is a clean 400 now, pinned by a
  test that also asserts nothing was persisted.
- unshelve never restarted the guest shelve stopped: the record said
  ACTIVE over a stopped guest, and the state reconciler (whose job is
  exactly that correction) flipped it back to SHUTOFF within one 30s
  sweep — 'server unshelve' observably did nothing, and the conformance
  case failed whenever the sweep won the race. Unshelve now starts the
  guest BEFORE writing ACTIVE (winning the reconciler's compare-and-set
  by construction: a SHELVED row is never a candidate), self-healing
  via reviveGuest like os-start when the domain is gone. Pinned by
  TestUnshelveRestartsGuest; the full run now logs zero guest-off
  corrections where it logged one.
- Migration 077 makes volumes.project_id and ports.project_id NOT NULL:
  both are scanned into plain strings by the ownership labels round 8
  added, and no writer can produce NULL — which is exactly why the
  invariant belongs in the schema. One metadata test fixture inserted
  a NULL port directly; it binds the seeded project now.
- Harness polish from the same round: a half-started compose stack no
  longer outlives a failed startup (teardown before the exit 2 — the
  trap is not installed yet at that point), the zero-suite error no
  longer interpolates an array that is necessarily empty, the cache
  tests' Redis cleanup actually matches its keys (DeletePattern
  prepends the prefix — a prefixed pattern doubled it and matched
  nothing), and neutron's dead 'networks:*' eviction patterns are gone
  (only network:<project>:<id> keys are ever written).

Verified at this HEAD: full run 249 pass / 0 fail / 2 known-failing /
0 skipped, exit 0, zero reconciler corrections; coverage gate PASSED;
full internal Go suite green.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
…rtial-up teardown

Round 10 reviewed the round-9 fixes and found one medium plus two small
follow-ups, all fixed:

- The unshelve SELF-HEAL left the row naming the dead domain while the
  revived guest ran under a new one: the reconciler then selected the
  row, asked about the stale id, and flipped ACTIVE back to SHUTOFF
  within one sweep — the exact bug the round-9 restart exists to fix,
  reborn on the self-heal path — and the next os-start self-heal could
  undefined the running guest in real mode (DomainCreate on an
  already-active domain fails and the error path undefines it). The
  ACTIVE update now writes the revived domain id, the way os-start's
  persistRunState always did. Pinned by TestUnshelveSelfHealPersistsRevivedDomain
  (shelve → guest vanishes → unshelve → the row's domain names a RUNNING
  guest), seeded with a root volume so the heal has stored state.
- Migration 078 extends 077's NOT NULL hardening to networks, routers
  and floating_ips — the remaining tables whose project_id is scanned
  into a plain string and rendered as tenant_id by the cross-project
  reads. Every writer binds the auth-context project (verified), so the
  constraint makes an existing invariant structural.
- run.sh marks the compose stack as started BEFORE 'up -d', not after:
  a partially-started project (up fails midway) previously left
  containers and ports behind — the failed-startup exit path only
  tears down a marked stack, and 'down -v' on an unstarted project is a
  harmless no-op.

Accepted as pre-existing (reviewer: informational): a shelve→unshelve
issued inside shelve's async stop window can still end SHUTOFF (upstream
serializes per-instance actions); a client disconnect mid-start leaves
the record SHELVED with the guest up, which a retry resolves.

Verified at this HEAD: full run 249 pass / 0 fail / 2 known-failing /
0 skipped, exit 0; coverage gate PASSED; full internal Go suite green.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
…ny/IPv6 allocation

Review findings (comprehensive review of the open senolcolak PRs):

- MEDIUM: nightly --strict was red on arrival because two known-failing cases
  are structurally unpassable (stub console 501; OSC 7.0.0 backup-show bug).
- MEDIUM: the planned '>0 new known-failing blocks' guard did not exist, so a
  regression on any of the ~26 self-healing probes passed both gates. Add a
  checked-in baseline (test/cli/baseline/known_failures.txt); coverage.sh and
  run.sh --strict now fail on any known-failing not in it and warn on stale
  entries. Live: --fast --strict 244/242 pass, full --strict 251/249 pass,
  2 baseline known-failing, exit 0; an empty baseline fails with both
  violations.
- MEDIUM: List/GetSubnetPool rendered the caller's project as tenant_id for
  shared pools; both now select and render the row's owning project.
- MEDIUM: allocateIP refused /30, /29 and all IPv6 pools as 'untenable'.
  Upstream Neutron allocates from them and fails only on exhaustion; the
  allocator now mirrors generate_pools() and never panics. Tests rewritten
  to assert allocation and a 201 from a tiny and an IPv6 pool.
- Numbering: documents (does not change) the SPEC-028 / migration-074
  collision with #108/#109 and the missing spec.md/tasks.md.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
senolcolak pushed a commit that referenced this pull request Nov 2, 2026
Review findings (comprehensive review of the open senolcolak PRs):

- MEDIUM: CLAUDE.md gave Cinder an S3 backend and listed rbd,s3 as valid;
  only Glance supports S3 and rbd,s3 is rejected at startup.
- MEDIUM: CLAUDE.md asserted '100% Terraform compatibility' and 'zero
  modifications', contradicted by the repo's own compatibility report and
  blueprint. Hedge to the validated topology and link the report.
- MEDIUM: 'next free spec number is 028' collides with in-flight #107 and
  #108/#109. Freeze 028, name the claimants, and keep the later-merge
  renumber rule (no renumbering here). Flag #107's plan.md-only spec.
- LOW: state Go 1.27.x and defer to go.mod, so #107's 1.27.3 bump does not
  instantly stale the floor.
- LOW: registry PR citations #515-#549 are cloud-storage PRs, not this
  repo's; prefix them. SPEC-019 cites commit 0a4dde0.
- LOW: AGENTS.md pointed the DB tests at the shared postgres database;
  standardize on lightstack_test.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
senolcolak pushed a commit that referenced this pull request Nov 2, 2026
Shared CI unblocker for the open PRs (independent of this PR's content):

- govulncheck now reports GO-2026-6809 in crypto/x509 (reachable through
  the HTTP server), fixed in Go 1.27.3. Main pins 1.27.2 everywhere, so
  every PR's Runtime Vulnerabilities and SBOM job fails. Pin 1.27.3 in
  go.mod, tools/go.mod, CI workflows, the builder image, the test
  container and the single-node deploy script (version + sha256).
- The unpinned Helm install now pulls Helm 4, whose 'helm plugin install'
  rejects the helm-unittest source without --verify=false, failing the
  Helm Charts job with 'plugin source does not support verification'.
  Install Helm 3.22.1 and helm-unittest 1.2.1 in an isolated plugin dir,
  matching PR #107, so the gate is version-stable.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
senolcolak pushed a commit that referenced this pull request Nov 2, 2026
Shared CI unblocker for the open PRs (independent of this PR's content):

- govulncheck now reports GO-2026-6809 in crypto/x509 (reachable through
  the HTTP server), fixed in Go 1.27.3. Main pins 1.27.2 everywhere, so
  every PR's Runtime Vulnerabilities and SBOM job fails. Pin 1.27.3 in
  go.mod, tools/go.mod, CI workflows, the builder image, the test
  container and the single-node deploy script (version + sha256).
- The unpinned Helm install now pulls Helm 4, whose 'helm plugin install'
  rejects the helm-unittest source without --verify=false, failing the
  Helm Charts job with 'plugin source does not support verification'.
  Install Helm 3.22.1 and helm-unittest 1.2.1 in an isolated plugin dir,
  matching PR #107, so the gate is version-stable.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
senolcolak pushed a commit that referenced this pull request Nov 2, 2026
Shared CI unblocker for the open PRs (independent of this PR's content):

- govulncheck now reports GO-2026-6809 in crypto/x509 (reachable through
  the HTTP server), fixed in Go 1.27.3. Main pins 1.27.2 everywhere, so
  every PR's Runtime Vulnerabilities and SBOM job fails. Pin 1.27.3 in
  go.mod, tools/go.mod, CI workflows, the builder image, the test
  container and the single-node deploy script (version + sha256).
- The unpinned Helm install now pulls Helm 4, whose 'helm plugin install'
  rejects the helm-unittest source without --verify=false, failing the
  Helm Charts job with 'plugin source does not support verification'.
  Install Helm 3.22.1 and helm-unittest 1.2.1 in an isolated plugin dir,
  matching PR #107, so the gate is version-stable.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
senolcolak pushed a commit that referenced this pull request Nov 2, 2026
Shared CI unblocker for the open PRs (independent of this PR's content):

- govulncheck now reports GO-2026-6809 in crypto/x509 (reachable through
  the HTTP server), fixed in Go 1.27.3. Main pins 1.27.2 everywhere, so
  every PR's Runtime Vulnerabilities and SBOM job fails. Pin 1.27.3 in
  go.mod, tools/go.mod, CI workflows, the builder image, the test
  container and the single-node deploy script (version + sha256).
- The unpinned Helm install now pulls Helm 4, whose 'helm plugin install'
  rejects the helm-unittest source without --verify=false, failing the
  Helm Charts job with 'plugin source does not support verification'.
  Install Helm 3.22.1 and helm-unittest 1.2.1 in an isolated plugin dir,
  matching PR #107, so the gate is version-stable.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
@senolcolak

Copy link
Copy Markdown
Collaborator Author

Comprehensive review — findings fixed

Reviewed against origin/main @ e909227, with a live run in an isolated netns. The core deliverable is strong: --fast 244/242 pass/0 fail, coverage.sh PASSED, zero teardown residue across 15 resource kinds, zero 500s server-side, migrations 074–078 round-trip with a byte-identical schema.

Found 8 (4 Medium, 4 Low) — pre-merge items fixed in 8397cff.

Severity Finding Fix
Med Nightly --strict was red on arrival: 2 known-failing cases are structurally unpassable (stub console 501; OSC 7.0.0 backup-show bug) Checked-in baseline test/cli/baseline/known_failures.txt; --strict passes baseline entries, fails new ones
Med The planned ">0 new known-failing blocks" guard did not exist — a regression on any of ~26 self-healing probes passed both gates silently coverage.sh/report now fail on any known-failing not in the baseline, warn on stale entries; harness tests cover all four paths. Live: full --strict 251/249 pass, 2 baseline, exit 0; empty baseline fails with both violations
Med List/GetSubnetPool rendered the caller's project as tenant_id for shared pools Both select and render the row's owning project; DB test with two projects
Med allocateIP refused /30, /29 and all IPv6 pools as "untenable" Upstream Neutron allocates from them and fails only on exhaustion — allocator now mirrors generate_pools(), never panics; tests assert 201 + non-blank IP from tiny and IPv6 pools
Low SPEC-028 claimed by three PRs; this spec dir is plan.md-only Documented only (no renumbering per maintainer): merge-dependency section names #108/#109 and the second-merge renumber rule
Low Token revoke is still a no-op (now with a 404 shape) Not fixed — needs a jti denylist or a documented limitation; flagged
Low 1-in-5 parallel neutron test flake (pre-exists on main) / make test-cli can validate a stale server Not fixed — follow-ups, not blockers

Merge dependency: migrations 074–078 collide with #108/#109's 074_reservation_node; the second merge must renumber (a golang-migrate duplicate version otherwise fails at startup). #104/#107 also conflict in ports.go/volumes.go and need a semantic resolution (see #104).

CI: 10/10 green, including CLI Conformance and the shared Go 1.27.3/Helm pin.

Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
senolcolak added a commit that referenced this pull request Nov 12, 2026
* fix(keystone): authorize the identity management surface

Comprehensive review 2026-10-13, findings C-1/C-2/C-3/H-4/M-5 (also found
independently as N1/N2/N5): the identity management API was
authentication-only. Every route under /v3 (except /auth/*) honored any
valid token, so a plain member could:

- reset any user's password — including the admin's — via PATCH
  /users/:id (which, unlike ChangePassword, never asks for the original)
  [C-1/N1, account takeover]
- grant itself the admin role via PUT /projects/:id/users/:uid/roles/:rid
  and revoke anyone else's [C-2/N2, privilege escalation]
- dump every tenant's recoverable EC2/S3 secret via GET /credentials,
  and mint/overwrite/delete credentials for other users [C-3]
- mint unrestricted application credentials for any user, receiving the
  plaintext secret in the 201 [H-4/N5]
- enumerate users, projects, roles, assignments, groups, domains,
  services, endpoints [M-5]

Fix: the same self-or-admin check the EC2-credentials API and
ChangePassword already use, plus an admin-only variant, applied across
the surface (new internal/keystone/authz.go helpers):

- users: List/Create admin-only; Get/GetProjects/GetGroups self-or-admin;
  PATCH self-or-admin, and password/enabled stay admin-only even for
  self (ChangePassword keeps its original-password contract)
- role assignment: Assign/Unassign/List/Check admin-only, with existence
  checks for project/user/role (the old code 204'd assignments
  referencing nonexistent rows)
- projects, groups, roles, role assignments, domains, services,
  endpoints: admin-only management; catalog reads admin-only
- generic credentials: List self-scoped for members (admin sees all),
  Create self-or-admin, Get/Update/Delete owner-or-admin
- application credentials: all four user-scoped routes self-or-admin;
  the id-only lookup 404s for foreign owners (no existence leak)

Convention kept (cloud-storage#515, TestIdentityAuthzPrecedence): gated
routes with path ids moved from the uuid-validating 'val' group to
'mgmt' and validate ids in-handler AFTER the authz check, so 403
precedes the verbatim 404. routes_wiring_test and
identity_notfound_test updated to the new wiring (admin caller, same 404
shapes); new identity_authz_gates_test pins 19 member denials, the
self-service allowances, and the unchanged admin path.

Unit suite (incl. DB-backed): PASS. Contract suite: no regressions (the
5 failures in this environment are the docker-exec-dependent tests that
cannot pass without docker, unchanged from baseline).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix(glance): image-poisoning, checksum-of-stored-bytes, task/cache authz, upload caps

Comprehensive review 2026-10-13 (H-1, M-1, M-2, M-3, N-3):

H-1 — upload authorization was visibility-based: the gate was
  `visibility = 'public' OR project_id = caller`, so any tenant that could
  SEE a public image could overwrite its bytes while queued — image
  poisoning of an operator-staged base that every tenant boots. Now the
  same owner-or-admin rule delete already uses (imageDeleteAuthz):
  403 for a visible-but-foreign image, 404 for an invisible one, admin
  override. Pinned by TestUploadImageDataOwnership_DB.

N-3 — the recorded checksum was MD5 of the upload stream, but normalizing
  backends (local, local,rbd, local,s3) rewrite the bytes (qcow2 → raw
  base): MD5(raw replica) != MD5(original qcow2), so EnsureLocalBase's
  integrity check failed on every node without the local base and
  boot-from-image errored — the exact multi-node case local,s3 exists
  for. New ImageStore.StoredChecksum hashes the stored object, and
  UploadImageData records it whenever the backend normalizes (stream
  hash stays correct for as-is backends: s3, stub). Pinned by
  TestStoredChecksumMatchesStoredBytesNotUploadBytes (real qemu-img,
  real qcow2).

M-1 — glance tasks leaked cross-tenant input/result/message and echoed
  internal errors: ListTasks/GetTask are now owner-scoped (admin sees
  all), GetTask 404s for foreign tasks, and no debug_error / err.Error()
  in response bodies.

M-2 — cache management (prefetch/evict/clear) is admin-only (upstream
  admin_api): any member could flush the deployment-wide cache with one
  request. Pinned by TestImageCacheManagementAdminOnly_DB.

M-3 — unbounded bodies: PUT /images/:id/file and POST stage now wrap
  the request in http.MaxBytesReader with a 10 GiB cap (upstream
  Glance's image_size_cap default).

Unit suite (incl. DB-backed): PASS. Contract suite: unchanged (only the
5 docker-exec-dependent tests fail, same as baseline).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix(nova): flavor admin surface, port lifecycle, attach/detach atomicity

Comprehensive review 2026-10-13 (H-2, N-6, N-8, N-9, N-10, L-3, L-4):

H-2 — the flavor catalog was member-writable: create/delete flavor,
  extra-specs CRUD, and tenant access grants (FlavorAction) all honored
  any authenticated caller. The catalog is a shared deployment-wide
  resource; upstream gates all of it admin_api. Now requireAdmin on all
  seven handlers. A member could resize the world's flavors out from
  under every tenant, or grant itself another project's private flavor.

L-4 — flavor inputs get upper sanity bounds (1 TiB RAM, 4096 vCPUs,
  1 PiB disk) instead of only >= 0: a typo (MB vs GB) used to schedule
  absurd values that surfaced only at boot time.

N-9 — AttachVolume was check-then-act across two statements: the loser
  of a concurrent double-attach surfaced as a 500 from the UNIQUE
  constraint, and a failure between the attachment INSERT and the volume
  UPDATE left the row inconsistent. Now one transaction with the volume
  row locked (FOR UPDATE), mirroring cinder's attach CAS; the concurrent
  double-attach answers 409, the state refusal stays 400.

N-10 — DetachVolume had the mirror problem: DELETE then UPDATE as two
  statements, so a failure after the DELETE stranded the volume in-use
  with no attachment row — unfixable through the API. Now one
  transaction; both statements or neither.

N-6 — a multi-network boot that failed on a later network leaked the
  ports already allocated for the earlier ones. The boot goroutine now
  releases what it allocated on every failure path after the first
  successful allocation (new NeutronService.ReleaseInstancePorts:
  deletes the instance's compute:nova ports, their TAP/FDB plumbing and
  DHCP reservations). Pinned by TestFailedBootReleasesAllocatedPorts /
  TestSuccessfulBootKeepsItsPorts.

N-8 — DeleteServer used a blanket DELETE of every port with the
  server's device_id, destroying ports the TENANT created and attached
  (os-interface with port_id) along with the boot-allocated ones.
  Upstream semantics now: nova-owned ports are deleted with the server,
  tenant-created ports are detached (device cleared, parked DOWN) and
  survive for their owner. Pinned by
  TestDeleteServerDetachesTenantPortsButDeletesItsOwn.

L-3 — CreateKeypair raced check-then-insert into duplicate-key 500s;
  the UNIQUE(user_id, name) violation now maps to the same 409 as the
  pre-check. Imported public keys are also bounded (64 KiB) before
  being parsed and stored (M-3).

Unit suite (incl. DB-backed): PASS. Contract suite: unchanged (only the
5 docker-exec-dependent tests fail, same as baseline).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix(cinder): admin-gate volume types, QoS specs and the manage surface; constant-time transfer keys

Comprehensive review 2026-10-13 (H-3, L-1):

H-3 — volume types, QoS specs and manage/unmanage were
  authentication-only. Any member could create volume types every
  tenant then booted from, rewrite extra specs, add its project to a
  private type (VolumeTypeAction), define QoS specs, and import host
  backings into its own quota (os-volume-manage addresses operator
  storage). All admin-only now, matching upstream's
  volume_extension:* and volume_admin_actions:* policies; reads of the
  type catalog stay open since tenants need them to pick a type at
  create. Pinned by TestOperatorSurfacesAdminOnly; the concurrent
  managed-import quota test drives the route as an administrator now
  (quota enforcement itself unchanged).

L-1 — the volume-transfer auth key was compared with a plain string
  equality, which short-circuits on the first differing byte; a
  transfer id is cheap to enumerate, so the key could be recovered
  byte-by-byte from response timing. Now crypto/subtle.
  ConstantTimeCompare.

Unit suite (incl. DB-backed): PASS.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix(neutron,metadata): collision-free IP allocation, shared-network gate, working network_data.json

Comprehensive review 2026-10-13 (N-4, M-4, N-7):

N-4 — port IP allocation was a purely random draw from the subnet, so
  two ports in one subnet landed on the same address at birthday-
  problem rates: two MACs claiming one IP, dnsmasq handing it to one of
  them, and a guest that could not reliably reach anything. New
  Service.allocateFreeIP draws random candidates until one is free of
  every port's fixed_ips AND the subnet's gateway, used by both
  CreatePort and AllocatePortForInstance; a full subnet is an honest
  refusal instead of a duplicate. Adjacent defect found and fixed while
  in here: allocateIP's candidate range [10, maxOffset) goes negative
  for subnets with fewer than ~12 addresses, and crypto/rand.Int
  PANICS on a non-positive bound — tiny subnets now fall back to the
  whole usable range. Pinned by TestAllocateFreeIP* (4 tests).

M-4 — CreateNetwork accepted shared=true from anyone, letting a member
  open their network to every project in the deployment (and every
  project's guests to it). Now gated exactly like router:external on
  create: only true is refused to a member, so clients echoing the
  default keep working (upstream's explicit-set rule). UpdateNetwork
  does not accept shared at all, so no gap there. Pinned by
  TestNonAdminCannotCreateSharedNetwork.

N-7 — the metadata service's network_data.json selected p.fixed_ip, a
  column that does not exist (ports store fixed_ips as a JSONB array),
  so the query failed and every guest fetching it got a 500 — cloud-init
  fell back to DHCP or configured nothing. Rewritten to resolve each
  port's addresses through the JSONB and their own subnets: one link
  per port (not per address, and not fanned out per subnet-on-network
  as the old join would have), subnet-resolved netmask/gateway/DNS, and
  a link even for an address-less port. Pinned by
  TestNetworkDataJSONServesPortAddresses.

Race-enabled unit suite (incl. DB-backed): PASS.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* docs(changelog): record the 2026-10-13 security and correctness fixes

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix: round-1 review findings on the authz-and-correctness PR

Findings from the review of this PR's own diff, all fixed:

HIGH — the N-8 tenant-port fix was unreachable through the real flow:
  os-interface stamped device_owner='compute:nova' on EVERY port it
  attached, including a tenant's pre-existing port_id, so every attached
  port looked nova-owned and server delete destroyed it anyway. The
  stamp now keys on which path ran: a port Nova creates (net_id) is
  nova's and deleted with the server; a tenant's port keeps its own
  owner and is detached. Mirrors upstream deallocate_for_instance,
  which skips ports it did not create. Pinned end-to-end by
  TestDeleteServerDetachesTenantPortsButDeletesItsOwn (attach via
  os-interface, then delete) and the new net_id companion
  TestDeleteServerDeletesInterfaceAttachCreatedPort.

MEDIUM — os-snapshot-manage missed by the H-3 sweep: admin-gated like
  os-volume-manage (upstream volume_admin_actions:snapshot_manage).

MEDIUM — volume types with a NULL description vanished from
  GET /v3/types (scan into a plain string failed and the row was
  silently dropped). COALESCE'd; pinned by
  TestListVolumeTypesServesNullDescription. The authz test is also
  idempotent now (unique names + cleanup; it would fail a second run
  against the shared test database).

MEDIUM — os-interface's net_id path wrote the NETWORK id into its
  fixed_ips subnet_id, so every consumer that joins on the subnet —
  network_data.json (N-7), DHCP, the N-4 collision check — silently
  missed those ports; and it ran its own next-free IP allocator
  (allocateNextIP), a second allocator over the same subnet that could
  hand out an address another path held. It now resolves the real
  subnet id and allocates through neutron.AllocateFreeIP (exported for
  exactly this); allocateNextIP is deleted. The port_id path also
  reported a hardcoded placeholder IP in its response — it now parses
  the port's actual first fixed IP.

MEDIUM — glance cache LIST was ungated while the rest of the cache
  surface was admin-only. Gated (upstream cache_list).

LOW — nova double-attach now answers 400 like upstream, not 409.
LOW — over-cap image uploads answer 413 (upstream image_size_cap
  semantics), not 503/400; new common.NewRequestEntityTooLargeError.
LOW — network_data.json port ordering gets a deterministic tiebreaker
  (created_at, id).
LOW — the boot test no longer writes under /var/lib: the node-local
  state root is redirected via paths.SetBase(t.TempDir()).

Unit suite (incl. DB-backed): PASS. Contract suite: unchanged (only the
5 docker-exec-dependent tests fail, same as baseline).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix: round-2 review findings — os-interface port_id path, upload-cap 413

Round 2 of the review loop (0 High, 1 Medium, 4 Low — all fixed):

MEDIUM (F1) — the port_id path of os-interface scanned the ports row's
  nullable device_id into a plain string. Every free port has NULL
  there, so the scan FAILED on every successful attach; the error was
  dropped and the attach worked only because pgx fills earlier columns
  before the failed one — and any real database error on that lookup
  surfaced as a bogus 200 with an empty net_id and MAC. The lookup now
  reads COALESCE(device_id::text, '') and handles its errors like the
  instance lookup above it. Pinned by
  TestAttachInterfacePortIDReportsRealState (attach a device_id-NULL
  port, assert the response carries the port's real net_id, MAC and
  fixed_ips).

LOW (F2) — the attach response fabricated one {"ip_address": ""} for
  a port with no addresses and omitted subnet ids. It now reports the
  port's actual fixed_ips entries (or an empty list). Pinned by
  TestAttachInterfacePortIDReportsRealState and
  TestAttachInterfacePortIDWithoutAddressesReportsNone.

LOW (F3) — an explicit fixed_ip on the net_id path was stored
  unvalidated: not checked against the subnet, not against the
  addresses other ports hold (the explicit side of the N-4 collision
  rule), and arbitrary text reached the JSONB build (a quote 500'd on
  the cast). Now: parse + in-subnet (400), availability via the new
  exported neutron.IPAvailable (409, upstream's 'IP address already
  allocated'). AllocateFreeIP and IPAvailable share one
  usedSubnetAddresses helper. Pinned by
  TestAttachInterfaceExplicitFixedIPIsValidated.

LOW (F4) — uploadImageLocal applied the 5-second metadata-op timeout
  to the body copy, so an over-cap upload hit the deadline (503) long
  before the 10 GiB MaxBytesReader cap could answer 413. The copy now
  runs under the caller's context only (cancelled on client
  disconnect), matching the S3 path's connect-only timeout policy
  (FR-014). maxImageUploadBytes is a var now, and
  TestOverCapUploadAnswers413 pins the 413 end-to-end (image left
  queued, not stuck saving).

LOW (F5) — the boot test's paths.SetBase restore could run while the
  boot goroutine was still reading the paths global (LIFO cleanup ran
  the restore before the service Shutdown). The redirect is now
  registered before newBootService so Shutdown stops the goroutine
  first.

Unit suite (incl. DB-backed): PASS. Contract suite: unchanged (only
the 5 docker-exec-dependent tests fail, same as baseline).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix(nova): round-3 review follow-ups — os-interface list and detach

Two LOW pre-existing findings surfaced by round 3 of the review loop,
in the same file and class as the round-2 fixes:

L-1 — ListInterfaceAttachments skipped any attachment whose port has no
  addresses, so an attachment created a moment earlier was invisible in
  the list — disagreeing with the attach response, which (since F2)
  reports it with an empty fixed_ips. The list now emits every
  attachment, defaulting fixed_ips to [] (JSON null in the column
  included). Pinned by TestListInterfacesIncludesAddresslessAttachments.

L-2 — DetachInterface had the dropped-error NULL scan F1 fixed in the
  attach path: a detached port (device_id NULL) failed the scan, the
  error was dropped, and the 'not attached' 409 was right only by
  accident — a real database error also 409'd instead of 500-ing.
  Mirrors the F1 fix: COALESCE(device_id::text, ''), errors handled.

Round 3's verdict on the PR's own commits was CLEAN; these two were
pre-existing on main.

Unit suite (incl. DB-backed): PASS. Contract suite: unchanged (only the
5 docker-exec-dependent tests fail, same as baseline).

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* test(nova): assert the empty fixed_ips serialization in the os-interface list

Round 4's only observation: the list test pinned the attachment's
presence but not the [] serialization it must agree with the attach
response on.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* test(nova): skip the success-path boot test when qemu-img is absent

The CI unit-test runner does not install qemu-img; the test's premise
(a boot that reaches ACTIVE) requires overlay staging, which needs it.
Same guard as pkg/storage's requireQemuImg — the failure-path N-6
tests, which expect ERROR anyway, still run everywhere.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix(neutron): close the JSON-null allocation DoS; align allocation with upstream

Review findings (comprehensive review of the open senolcolak PRs):

- HIGH: usedSubnetAddresses scanned ports.fixed_ips with a bare
  jsonb_array_elements. A port on a subnet-less network persists JSON-scalar
  null, and one such row made every allocation fail deployment-wide
  (SQLSTATE 22023) with a misleading 409. Guard the SRF with jsonb_typeof and
  write [] instead of null; legacy rows are tolerated by the readers.
- MEDIUM: metadata network_data.json used COALESCE, which does not fire for
  JSON null; same guard. A guest on such a port got a 500.
- LOW: the same unguarded scan in the nova ?ip= server filter.
- LOW: gate /os-hypervisors, /os-hypervisors/detail, /os-hypervisors/stats,
  /os-services and /os-console-auth-tokens/{token} with requireAdmin;
  statistics leaked a real deployment-wide running-VM count to members.
- MEDIUM: allocateIP now mirrors upstream generate_pools() for /30, /31,
  /32 and IPv6 pools instead of overflowing int64 on IPv6; small pools
  allocate, only exhaustion fails.

Tests pin the DoS and the metadata 500 (each fails on the unfixed query), the
admin gates, and tiny/IPv6 allocation.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* ci: move to Go 1.27.3 (GO-2026-6809) and pin Helm 3.22.1

Shared CI unblocker for the open PRs (independent of this PR's content):

- govulncheck now reports GO-2026-6809 in crypto/x509 (reachable through
  the HTTP server), fixed in Go 1.27.3. Main pins 1.27.2 everywhere, so
  every PR's Runtime Vulnerabilities and SBOM job fails. Pin 1.27.3 in
  go.mod, tools/go.mod, CI workflows, the builder image, the test
  container and the single-node deploy script (version + sha256).
- The unpinned Helm install now pulls Helm 4, whose 'helm plugin install'
  rejects the helm-unittest source without --verify=false, failing the
  Helm Charts job with 'plugin source does not support verification'.
  Install Helm 3.22.1 and helm-unittest 1.2.1 in an isolated plugin dir,
  matching PR #107, so the gate is version-stable.

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>

* fix(keystone): bind S3 credentials to authorized project scope

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix: serialize IP and volume device allocation

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix(glance): bound staging memory and fail incomplete uploads

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix: align guest network metadata and flavor bounds with Flamingo

Signed-off-by: Senol Colak <senol.colak@sap.com>

* docs: record PR 104 follow-up review and validation

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix: correct merge adaptations for SQLite and Helm CI

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix(nova): use database unique violation helper

Signed-off-by: Senol Colak <senol.colak@sap.com>

* refactor(nova): remove obsolete IP allocator

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix: repair PR 104 unit test failures

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix: address PR 104 review feedback

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix(keystone): normalize authorized uuid path params

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix: resolve remaining PR 104 review threads

Signed-off-by: Senol Colak <senol.colak@sap.com>

* fix(neutron): remove unused bytes import

Signed-off-by: Senol Colak <senol.colak@sap.com>

---------

Signed-off-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Co-authored-by: Uwe Schwaeke <uwe.schwaeke@sap.com>
Preserve serialized subnet allocation and port security-group behavior. Keep nullable Cinder descriptions and extra-spec responses. Use setup-go caching exclusively to avoid overlapping module-cache restores.

Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
Signed-off-by: Senol Colak <senol.colak@sap.com>
@senolcolak
senolcolak requested a review from uw-sc November 13, 2026 12:09

@uw-sc uw-sc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together. The CLI conformance suite, and the fixes it surfaced, are a real step forward.

Before this can merge, it needs some changes. The details, with tested suggestions, are in the inline comments. The most important are:

  • the admin lookups by name
  • the application-credential roles
  • the two CI gaps: the Redis service and the Terraform wrapper
  • the tier-B negative checks in the suites

Happy to take another look once those are addressed.

Comment on lines +55 to +58
# THE POINT OF THIS FILE: test the binary this checkout builds. The
# image's own copy is shadowed; CI downloads the build artifact into
# bin/ before starting this stack.
- ../bin/lightstack:/app/lightstack:ro

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make build produces a dynamically linked glibc binary (cgo is on by default; file bin/lightstack shows interpreter /lib64/ld-linux-x86-64.so.2). The runtime image is alpine:3.24 (musl, no gcompat), so the mounted binary fails to start with no such file or directory. Without a running server, make test-cli waits out the readiness timeout and fails.

Two more things keep this stack from testing the checkout:

  • The binary migrates from the image's own /app/migrations and reads the image's config.
  • up -d reuses an existing lightstack-binary-test:latest without rebuilding it.

CI doesn't use this file (the cli-conformance job runs the artifact natively), so the comment below is out of date as well.

Suggested fix: build statically in the Makefile's build target (CGO_ENABLED=0 go build …, as build/package/Dockerfile already does), and mount this checkout's migrations and config:

Suggested change
# THE POINT OF THIS FILE: test the binary this checkout builds. The
# image's own copy is shadowed; CI downloads the build artifact into
# bin/ before starting this stack.
- ../bin/lightstack:/app/lightstack:ro
# THE POINT OF THIS FILE: test the binary this checkout builds, against this
# checkout's migrations and config (the image's copies are shadowed). The
# binary must be static (CGO_ENABLED=0): the runtime image is Alpine (musl).
- ../bin/lightstack:/app/lightstack:ro
- ../migrations:/app/migrations:ro
- ../config/lightstack.yaml:/app/config/lightstack.yaml:ro

Nothing uses the Postgres "5433:5432" mapping (the stack reaches Postgres over the compose network), so it can go. Alternatively, drop the fallback and require a running server, as CI does.

Comment on lines +1043 to +1054
query := `
SELECT id, name, admin_state_up, status, shared, mtu, created_at, updated_at, project_id, external, COALESCE(description, '')
FROM networks
WHERE (id::text = $1 OR name = $1) AND `+networkReadScope("$2")+`
LIMIT 1
`, networkID, projectID).Scan(&id, &name, &adminStateUp, &status, &shared, &mtu, &createdAt, &updatedAt, &owner, &external)
WHERE (id::text = $1 OR name = $1)`
args := []any{networkID}
if !common.HasRole(c, common.RoleAdmin) {
query += ` AND ` + networkReadScope("$2")
args = append(args, projectID)
}
query += ` LIMIT 1`

err := svc.pool.QueryRow(ctx, query, args...).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an administrator this lookup drops the project scope entirely. The same query also matches by name, so an admin's name lookup now matches every project's networks (LIMIT 1, no ORDER BY). openstacksdk's find_network() tries GET /v2.0/networks/<name_or_id> before its ?name= fallback, and UpdateNetwork already exempts admins from the owner predicate. As a result, openstack network set --description x private run by admin can rewrite another project's network called private.

Reproduced on PostgreSQL and SQLite with a private network that exists only in another project. As admin of default:

  • GET /v2.0/networks/<its name> returns 200 with that network.
  • PUT /v2.0/networks/<returned id> returns 200, and the other project's description changes.

Upstream has no by-name GET, and its list fallback raises on duplicate names instead of picking one. CLAUDE.md applies the same rule to servers: a name never resolves outside the caller's own project.

Suggested fix: keep the carve-out for the id only, and bind fixed arguments:

Suggested change
query := `
SELECT id, name, admin_state_up, status, shared, mtu, created_at, updated_at, project_id, external, COALESCE(description, '')
FROM networks
WHERE (id::text = $1 OR name = $1) AND `+networkReadScope("$2")+`
LIMIT 1
`, networkID, projectID).Scan(&id, &name, &adminStateUp, &status, &shared, &mtu, &createdAt, &updatedAt, &owner, &external)
WHERE (id::text = $1 OR name = $1)`
args := []any{networkID}
if !common.HasRole(c, common.RoleAdmin) {
query += ` AND ` + networkReadScope("$2")
args = append(args, projectID)
}
query += ` LIMIT 1`
err := svc.pool.QueryRow(ctx, query, args...).
// The carve-out covers the id only: a NAME stays resolved within the caller's read scope, so a
// name another project also uses never reaches that project's network.
scope := networkReadScope("$2")
idScope := scope
if common.HasRole(c, common.RoleAdmin) {
idScope = "TRUE"
}
query := `
SELECT id, name, admin_state_up, status, shared, mtu, created_at, updated_at, project_id, external, COALESCE(description, '')
FROM networks
WHERE (id::text = $1 AND ` + idScope + `) OR (name = $1 AND ` + scope + `)
LIMIT 1`
err := svc.pool.QueryRow(ctx, query, networkID, projectID).

The same shape applies to GetSubnet (network.go:1851), GetRouter (router.go:301) and GetFloatingIP (floatingip.go:416, by address). With this change the scenario above answers 404, and internal/neutron stays green on both backends, including the carve-out tests, which address by id. Worth keeping the scenario as a regression test.

Comment on lines +1018 to +1027
getQuery := `
SELECT id, name, description, size_gb, status, bootable, user_id, volume_type, project_id, created_at, updated_at
FROM volumes
WHERE project_id = $2 AND ((id::text = $1) OR (name = $1))
`, volumeID, projectID).Scan(&id, &name, &description, &size, &status, &bootable, &userID, &volumeType, &createdAt, &updatedAt)
WHERE ((id::text = $1) OR (name = $1))`
getArgs := []any{volumeID}
if !common.HasRole(c, common.RoleAdmin) {
getQuery += ` AND project_id = $2`
getArgs = append(getArgs, projectID)
}
err := svc.pool.QueryRow(c.Request.Context(), getQuery, getArgs...).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue as in GetNetwork: for an administrator the name branch matches every project's volumes, so GET /v3/volumes/<name> returns another project's volume. Reproduced on PostgreSQL and SQLite; the CLI's find_resource tries this GET before its list fallback. The fix is to keep the carve-out for the id only:

Suggested change
getQuery := `
SELECT id, name, description, size_gb, status, bootable, user_id, volume_type, project_id, created_at, updated_at
FROM volumes
WHERE project_id = $2 AND ((id::text = $1) OR (name = $1))
`, volumeID, projectID).Scan(&id, &name, &description, &size, &status, &bootable, &userID, &volumeType, &createdAt, &updatedAt)
WHERE ((id::text = $1) OR (name = $1))`
getArgs := []any{volumeID}
if !common.HasRole(c, common.RoleAdmin) {
getQuery += ` AND project_id = $2`
getArgs = append(getArgs, projectID)
}
err := svc.pool.QueryRow(c.Request.Context(), getQuery, getArgs...).
// The carve-out covers the id only: a NAME stays resolved within the caller's project.
idScope := "project_id = $2"
if common.HasRole(c, common.RoleAdmin) {
idScope = "TRUE"
}
getQuery := `
SELECT id, name, description, size_gb, status, bootable, user_id, volume_type, project_id, created_at, updated_at
FROM volumes
WHERE (id::text = $1 AND ` + idScope + `) OR (name = $1 AND project_id = $2)`
err := svc.pool.QueryRow(c.Request.Context(), getQuery, volumeID, projectID).

internal/cinder stays green on both backends with this change.

Comment on lines +174 to +194
for _, role := range req.ApplicationCredential.Roles {
idStr, _ := role["id"].(string)
nameStr, _ := role["name"].(string)
var rid, rname string
var err error
switch {
case idStr != "":
err = svc.pool.QueryRow(c.Request.Context(),
"SELECT id::text, name FROM roles WHERE id = $1", idStr).Scan(&rid, &rname)
case nameStr != "":
err = svc.pool.QueryRow(c.Request.Context(),
"SELECT id::text, name FROM roles WHERE name = $1", nameStr).Scan(&rid, &rname)
default:
common.SendError(c, common.NewBadRequestError("each role must have an id or name"))
return
}
if err != nil {
common.SendError(c, common.NewBadRequestError("application credential references a role that does not exist"))
return
}
if !seenRoles[rid] {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upstream Keystone (stable/2025.2, keystone/api/users.py, _get_roles) handles requested roles differently:

  • A requested role that the creator's token doesn't carry is refused with 400: "Cannot create an application credential with unassigned role".
  • When the request names no roles, the credential gets the token's roles (roles = token.roles).

Here a role is only checked for existence. Any caller allowed to create the credential (the user themselves, or an admin) can attach any existing role, admin included. The row persists now that migration 076 creates the table. A request without roles yields "roles": []. Nothing authenticates with application credentials yet, so this isn't an escalation today; it becomes one once that auth method exists.

Suggested fix, mirroring _get_roles:

Suggested change
for _, role := range req.ApplicationCredential.Roles {
idStr, _ := role["id"].(string)
nameStr, _ := role["name"].(string)
var rid, rname string
var err error
switch {
case idStr != "":
err = svc.pool.QueryRow(c.Request.Context(),
"SELECT id::text, name FROM roles WHERE id = $1", idStr).Scan(&rid, &rname)
case nameStr != "":
err = svc.pool.QueryRow(c.Request.Context(),
"SELECT id::text, name FROM roles WHERE name = $1", nameStr).Scan(&rid, &rname)
default:
common.SendError(c, common.NewBadRequestError("each role must have an id or name"))
return
}
if err != nil {
common.SendError(c, common.NewBadRequestError("application credential references a role that does not exist"))
return
}
if !seenRoles[rid] {
// Upstream (keystone/api/users.py _get_roles): no roles means the creator's
// token roles, and a requested role the token does not carry is a 400.
requested := req.ApplicationCredential.Roles
if len(requested) == 0 {
for _, name := range c.GetStringSlice("roles") {
requested = append(requested, map[string]any{"name": name})
}
}
for _, role := range requested {
idStr, _ := role["id"].(string)
nameStr, _ := role["name"].(string)
var rid, rname string
var err error
switch {
case idStr != "":
err = svc.pool.QueryRow(c.Request.Context(),
"SELECT id::text, name FROM roles WHERE id = $1", idStr).Scan(&rid, &rname)
case nameStr != "":
err = svc.pool.QueryRow(c.Request.Context(),
"SELECT id::text, name FROM roles WHERE name = $1", nameStr).Scan(&rid, &rname)
default:
common.SendError(c, common.NewBadRequestError("each role must have an id or name"))
return
}
if err != nil {
common.SendError(c, common.NewBadRequestError("application credential references a role that does not exist"))
return
}
if !common.HasRole(c, rname) {
common.SendError(c, common.NewBadRequestError("Cannot create an application credential with unassigned role"))
return
}
if !seenRoles[rid] {

Verified on PostgreSQL: an unassigned role then answers 400, and a request without roles gets the token's roles. TestApplicationCredentialConformance and TestCredentialRolesAreAtomic pin the current behaviour, because they request a role the test token doesn't carry. They would need to request one it does, e.g. via X-Test-Roles or c.Set("roles", …). The comment at line 252 ("an empty array when none were requested") changes accordingly.

Comment on lines 71 to +98
@@ -90,8 +94,8 @@ func (svc *Service) ListApplicationCredentials(c *gin.Context) {
log.Warn().Err(err).Msg("roleRows iteration error; list may be truncated")
}
roleRows.Close()
credential["roles"] = roles
}
credential["roles"] = roles

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 25-line roles lookup appears three times: here, in GetApplicationCredential (line 318) and in GetApplicationCredentialByID (line 432). Each copy still swallows errors:

  • A failed query only logs a warning.
  • Scan errors are skipped (err == nil).
  • rows.Err() only warns.

Before this change a failed lookup dropped the key. Now it answers 200 with "roles": [], which is a definite and wrong statement about what the credential carries.

Suggested fix: one helper that returns the error, and a 500 at each call site:

Suggested change
roles, err := svc.credentialRoles(c.Request.Context(), id)
if err != nil {
log.Error().Err(err).Str("application_credential_id", id).Msg("failed to read application credential roles")
common.SendError(c, common.NewInternalServerError("failed to read application credential roles"))
return
}
credential["roles"] = roles

with the helper, e.g. at the end of the file (plus the context import):

// credentialRoles returns the roles an application credential carries — an empty slice when it
// has none, so the key always renders as upstream's array.
func (svc *Service) credentialRoles(ctx context.Context, id string) ([]map[string]any, error) {
	rows, err := svc.pool.Query(ctx, `
		SELECT r.id, r.name
		FROM application_credential_roles acr
		JOIN roles r ON acr.role_id = r.id
		WHERE acr.application_credential_id = $1`, id)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	roles := []map[string]any{}
	for rows.Next() {
		var roleID, roleName string
		if err := rows.Scan(&roleID, &roleName); err != nil {
			return nil, err
		}
		roles = append(roles, map[string]any{"id": roleID, "name": roleName})
	}
	return roles, rows.Err()
}

The same call replaces the copies at lines 318–345 and 432–459, which removes 39 lines. internal/keystone stays green on PostgreSQL and SQLite with this change (gofmt and vet clean). Behaviour is unchanged except that a database failure now answers 500.

Comment thread test/terraform/run.sh
Comment on lines +161 to +162
log "terraform version"
terraform version | head -1

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TF_VERSION_PIN appears only in the install hint (lines 146–150), so any terraform version runs, and nightly.yml repeats 1.13.4 separately. Enforcing the pin at the version step:

Suggested change
log "terraform version"
terraform version | head -1
tf_version="$(terraform version -json | jq -r .terraform_version)" \
|| die "cannot read the terraform version"
[ "$tf_version" = "$TF_VERSION_PIN" ] \
|| die "terraform $tf_version found; this wrapper is pinned to $TF_VERSION_PIN"
log "terraform $tf_version"

The fake terraform in test_runner.py prints plain text for version, so it needs a -json branch, e.g. print(json.dumps({"terraform_version": "1.13.4"})) when "-json" in args. With that, all seven wrapper tests pass, and a fake that reports 1.12.0 makes the run stop as intended.

Two more pieces do nothing today:

  • The header promises "Provider downloads are shared through TF_PLUGIN_CACHE_DIR so repeat runs (and the nightly CI job) don't re-download the provider". But line 219 deletes .terraform.lock.hcl, and the repo commits none. Without lock-file checksums Terraform doesn't take the provider from the cache unless TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE is set (internal/command/cliconfig/cliconfig.go), and the nightly has no cache step anyway.
  • The floating-IP capture (lines 271–281) and absence check (lines 318–332) target openstack_networking_floatingip_v2, which the compat module never creates (grep -c over test/terraform/compat/*.tf is 0). They only add a state pull failure mode, and could come back together with a floating-IP resource.

Comment on lines +68 to +72
TEST_OUTPUT=$(go test -json -count=1 -timeout=3m 2>&1) || TEST_RC=$?

# Count results. `|| true`: jq exits non-zero if go test mixed non-JSON
# stderr (e.g. build errors) into the stream; the counts we did get are
# still what we want to report, and zero matches is a normal outcome.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2>&1 mixes go's stderr into the JSON stream. On a cold module cache that means go: downloading … lines, and jq stops at the first non-JSON line: jq: parse error: Invalid numeric literal at line 1, column 3, exit 5, nothing counted. A passing package then reports 0/0 and is counted as "did not complete a passing test run", so the comment below assumes the opposite of what happens. With Go 1.27, go test -json reports build failures as JSON events on stdout, so stderr can simply stay out:

Suggested change
TEST_OUTPUT=$(go test -json -count=1 -timeout=3m 2>&1) || TEST_RC=$?
# Count results. `|| true`: jq exits non-zero if go test mixed non-JSON
# stderr (e.g. build errors) into the stream; the counts we did get are
# still what we want to report, and zero matches is a normal outcome.
TEST_OUTPUT=$(go test -json -count=1 -timeout=3m) || TEST_RC=$?
# Count results. stderr stays out of the stream: jq stops at the first
# non-JSON line, so a "go: downloading ..." line would zero the counts of
# a passing package. Zero matches is a normal outcome.

Separately, SERVICES=("keystone" "nova" "cinder" "neutron" "glance") never runs the root test/contract package. That package holds six real tests, among them TestNovaServerReadBack_KeyName_Contract and TestNovaServerReadBack_SecurityGroups_Contract, plus the WS0/WS4/WS5 ones, yet test/contract/README.md offers this script under "Run All Contract Tests". Running go test -json ./... once from test/contract and counting per .Package would cover everything, and would keep the next new package from being missed the way nova once was.

Comment thread test/cli/lib/report.sh
Comment on lines +138 to +157
# case_line: "<classname>\t<name>\t<status>\t<duration_ms>\t<command>\t<error>"
local cname name status dur cmd err
IFS=$'\t' read -r cname name status dur cmd err <<<"$case_line"
printf ' <testcase classname="%s" name="%s" time="%.3f"' \
"$(_xml_escape "$cname")" "$(_xml_escape "$name")" "$(awk -v d="${dur:-0}" 'BEGIN { printf "%.3f", d / 1000 }')"
if [[ "$status" == "fail" ]]; then
printf '>\n <failure message="%s">%s</failure>\n </testcase>\n' \
"$(_xml_escape "${cmd:-}")" "$(_xml_escape "${err:-assertion failed}")"
elif [[ "$status" == "known-failing" ]]; then
printf '>\n <skipped message="%s">%s</skipped>\n </testcase>\n' \
"$(_xml_escape "known failure: ${err:-}")" "$(_xml_escape "${err:-}")"
elif [[ "$status" == "skip" ]]; then
printf '>\n <skipped message="%s"/>\n </testcase>\n' \
"$(_xml_escape "${err:-skipped}")"
else
printf '/>\n'
fi
done < <(jq -r --arg s "$s" \
'select(.suite == $s) | [.suite, .id, .status, (.duration_ms | tostring), .command, .error] | @tsv' \
"$RESULTS_FILE")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IFS=$'\t' read collapses empty fields, because tab is IFS whitespace (IFS=$'\t' read -r a b c <<<$'x\t\ty' gives b=y). A failing case with an empty command therefore shifts its error into the wrong slot. With a crafted results file, this writer emits <failure message="boom: HTTP 500">assertion failed</failure> for such a case. A non-whitespace separator keeps every slot, and the same file then gives <failure message="">boom: HTTP 500</failure>:

Suggested change
# case_line: "<classname>\t<name>\t<status>\t<duration_ms>\t<command>\t<error>"
local cname name status dur cmd err
IFS=$'\t' read -r cname name status dur cmd err <<<"$case_line"
printf ' <testcase classname="%s" name="%s" time="%.3f"' \
"$(_xml_escape "$cname")" "$(_xml_escape "$name")" "$(awk -v d="${dur:-0}" 'BEGIN { printf "%.3f", d / 1000 }')"
if [[ "$status" == "fail" ]]; then
printf '>\n <failure message="%s">%s</failure>\n </testcase>\n' \
"$(_xml_escape "${cmd:-}")" "$(_xml_escape "${err:-assertion failed}")"
elif [[ "$status" == "known-failing" ]]; then
printf '>\n <skipped message="%s">%s</skipped>\n </testcase>\n' \
"$(_xml_escape "known failure: ${err:-}")" "$(_xml_escape "${err:-}")"
elif [[ "$status" == "skip" ]]; then
printf '>\n <skipped message="%s"/>\n </testcase>\n' \
"$(_xml_escape "${err:-skipped}")"
else
printf '/>\n'
fi
done < <(jq -r --arg s "$s" \
'select(.suite == $s) | [.suite, .id, .status, (.duration_ms | tostring), .command, .error] | @tsv' \
"$RESULTS_FILE")
# case_line: classname, name, status, duration_ms, command, error — joined
# by \x1f, which bash never collapses (an empty command keeps its slot).
local cname name status dur cmd err
IFS=$'\x1f' read -r cname name status dur cmd err <<<"$case_line"
printf ' <testcase classname="%s" name="%s" time="%.3f"' \
"$(_xml_escape "$cname")" "$(_xml_escape "$name")" "$(awk -v d="${dur:-0}" 'BEGIN { printf "%.3f", d / 1000 }')"
if [[ "$status" == "fail" ]]; then
printf '>\n <failure message="%s">%s</failure>\n </testcase>\n' \
"$(_xml_escape "${cmd:-}")" "$(_xml_escape "${err:-assertion failed}")"
elif [[ "$status" == "known-failing" ]]; then
printf '>\n <skipped message="%s">%s</skipped>\n </testcase>\n' \
"$(_xml_escape "known failure: ${err:-}")" "$(_xml_escape "${err:-}")"
elif [[ "$status" == "skip" ]]; then
printf '>\n <skipped message="%s"/>\n </testcase>\n' \
"$(_xml_escape "${err:-skipped}")"
else
printf '/>\n'
fi
done < <(jq -r --arg s "$s" \
'select(.suite == $s) | [.suite, .id, .status, (.duration_ms | tostring), .command, .error] | ([31] | implode) as $us | map(. // "" | tostring | gsub("\n"; " ") | gsub($us; " ")) | join($us)' \
"$RESULTS_FILE")

bash -n is clean with this. Three smaller things in the same plumbing:

  • The baseline verdict is implemented twice, at report.sh:236-265 and coverage.sh:158-193. Keeping it only in coverage.sh leaves one rule to maintain.
  • coverage.sh silently drops a manifest whose commands key is misspelled, or whose tiers is empty. Renaming the key in image.json makes its 14 entries vanish from the report without a warning. A jq check that every manifest has a non-empty commands array, and that every entry has an id and tiers within A/B/C/S, would fail closed.
  • ${TMPDIR:-/tmp}/lightstack-cli-err.$$ (common.sh:138) is a predictable name in a shared /tmp, so mktemp would be safer.

Comment on lines +513 to +519
t_begin keystone.role.assignment.remove.neg
# admin holds only the admin role on the default project; removing 'reader'
# names an assignment that does not exist
osc role remove reader --user admin --project default
_ke_assert_failed
_ke_assert_http 404
t_end

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This negative removes a role from the real admin user. On a server where admin does hold reader on default, the "negative" succeeds and deletes a real assignment, which nothing restores. The suite supports running against an existing server, so that case is reachable. A throwaway user and project keep the same check without touching anyone's assignments:

Suggested change
t_begin keystone.role.assignment.remove.neg
# admin holds only the admin role on the default project; removing 'reader'
# names an assignment that does not exist
osc role remove reader --user admin --project default
_ke_assert_failed
_ke_assert_http 404
t_end
t_begin keystone.role.assignment.remove.neg
# A throwaway user and project with no assignments: removing 'reader' names an
# assignment that cannot exist, and no real user's assignment is at stake.
osc user delete cli-conf-ke-rnu
osc project delete cli-conf-ke-rnp
osc user create cli-conf-ke-rnu --password ke-pass-123
assert_rc 0
osc project create cli-conf-ke-rnp
assert_rc 0
osc role remove reader --user cli-conf-ke-rnu --project cli-conf-ke-rnp
_ke_assert_failed
_ke_assert_http 404
osc user delete cli-conf-ke-rnu
osc project delete cli-conf-ke-rnp
t_end

The keystone suite stays 76/76 green with this (pinned OSC 7.0.0 against a stub-mode server), and the throwaway user and project are gone afterwards.

Related hygiene points:

  • workflows.sh:586 and :595 run role add … --user admin admin regardless of OS_USERNAME. --user "$OS_USERNAME" --user-domain "$OS_USER_DOMAIN_NAME" follows the configured runner.
  • compute.sh:21, image.sh:21, volume.sh:23 and workflows.sh:27 call fixtures_require without checking its status, so a standalone run continues on missing fixtures. fixtures_require || exit 2 stops it there.
  • _nt_clean_network (network.sh:93) never releases floating IPs, and a network that still has some refuses deletion (networkInUse, network.go:1191). After an interrupted run, cli-conf-nt-fipset-ext can't be removed, and the next run creates a second network of that name, whose --network lookup then fails as ambiguous. Releasing the network's floating IPs first avoids that.
  • The fixture sweep deletes subnets before routers and never detaches interfaces, while deleting a router that still has interfaces answers 409 (router.go:457). Leftover routers from a crashed run therefore survive every later sweep. Running _nt_clean_router from fixtures.sh before the subnet sweep would reclaim them.

Comment thread test/cli/lib/common.sh
Comment on lines +164 to +170
local args=("$@") has_format=0 a
for a in "$@"; do
[[ "$a" == "-f" || "$a" == "--format" ]] && has_format=1
done
# Machine-readable output unless the caller explicitly asked otherwise
# (e.g. -f value, -f csv, --help).
(( has_format )) || args+=(-f json)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

osc() appends -f json to every command that has no format flag. delete/set/unset commands have no formatter, so the CLI rejects the flag with argparse exit 2, and the wrapper retries without it. Each such call therefore costs two CLI starts and two token requests: cliff runs prepare_to_run_command, where osc_lib authenticates, before it parses the arguments (cliff/app.py:438 vs :446). The suites contain 351 such call sites (delete 295, set 51, unset 5), some of them in loops. Skipping the flag for those verbs up front leaves the retry as the fallback for everything else:

Suggested change
local args=("$@") has_format=0 a
for a in "$@"; do
[[ "$a" == "-f" || "$a" == "--format" ]] && has_format=1
done
# Machine-readable output unless the caller explicitly asked otherwise
# (e.g. -f value, -f csv, --help).
(( has_format )) || args+=(-f json)
local args=("$@") has_format=0 a
for a in "$@"; do
[[ "$a" == "-f" || "$a" == "--format" ]] && has_format=1
done
# delete/set/unset commands have no formatter and reject -f json with
# argparse exit 2; skip the flag for them rather than paying for a second
# CLI start and a second token request through the retry below.
for a in "${@:1:4}"; do
[[ "$a" == "delete" || "$a" == "set" || "$a" == "unset" ]] && has_format=1
done
# Machine-readable output unless the caller explicitly asked otherwise
# (e.g. -f value, -f csv, --help).
(( has_format )) || args+=(-f json)

The compute suite (pinned OSC 7.0.0 against a stub-mode server) records the same results with this, 63 passed and 1 baselined known-failing, in 299 s. Two runs without it took 376 s and 404 s.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement] Testing

2 participants